Notes for 2 April 2009 meeting 1000-1100 Bldg G Conference Room

Attending: Fred Bahr,  Kevin Gomes, Mike McCann

Review of THREDDS server data offerings: http://elvis.shore.mbari.org/thredds/catalog.html

• Mooring TS data are aggregated and available through the server
• For the examples below use the 9 April 2009 catalog: http://elvis.shore.mbari.org/thredds/catalog_090409.html
• Entire archive of decimated Data is available now via THREDDS.
• How applications can make use of the data is to be determined: Can Matlab & loaddap load and use the data?
  Not really:

  • Ferret cannot load all the aggregated irregular time data for a mooring:
    
    <103 elvis.shore.mbari.org /u/mccann> ferret
            NOAA/PMEL TMAP
            FERRET v6.13 (beta)
            Linux(g77) 2.4.21-32 - 08/28/08
             2-Apr-09 09:52
    
    cancel mode journal
    sp rm -f ferret.jnl
    
    yes? use "http://elvis.shore.mbari.org/thredds/dodsC/moorings/OS_M1_TS_090409"
     **TMAP ERR: limit on storage for coordinates has been reached
                 MAX=      750000

  • Matlab's loaddap also fails with an I/O error when trying to load all the mooring data:
    
    >> addpath('/usr/local/bin');     % For the loaddap command on elvis
    >> loaddap('http://elvis.shore.mbari.org/thredds/dodsC/moorings/OS_M1_TS_090409')
    
    Reading: http://elvis.shore.mbari.org/thredds/dodsC/moorings/OS_M1_TS_090409
      Constraint:
    Error: Network I/O error. Could not read packed array data.
    This may be due to a bug in libdap or a problem with
    the network connection.

  • However; a single variable (TEMP) can be loaded:
    >> loaddap('http://elvis.shore.mbari.org/thredds/dodsC/moorings/OS_M1_TS_090409?TEMP')
    
    Reading: http://elvis.shore.mbari.org/thredds/dodsC/moorings/OS_M1_TS_090409
      Constraint: TEMP
    Server version: apache-coyote/1.1
    Creating matrix TEMP (528789 x 11 x 1 x 1) with 5816679 elements.
    Creating vector TIME with 528789 elements.
    Creating vector DEPTH with 11 elements.
    Creating vector LATITUDE with 1 elements.
    Creating vector LONGITUDE with 1 elements.
    
    and plotted against epoch seconds:
    >> T=squeeze(TEMP.TEMP);   % Remove singleton lat & lon dimensions
    >> bindx=find(T < 0);      % Find missing values
    >> T(bindx)=NaN;           % Set them to NaN
    >> offset=datenum(1970,1,1,0,0,0);                  % Epoch second offset in Matlab's datenum
    >> plot(TEMP.TIME/24/60/60 + offset, T(1,:))        % Plot Surface temp for the whole time series
    >> datetick('x', 2)                                 % Give a reasonable time axis

  • loaddap fails to load the data with a stride specification (this looks like a bug):
    
    >> loaddap('http://elvis.shore.mbari.org/thredds/dodsC/auv/dorado_ctd?latitude[1:2:4000000]')
    Reading: [http://elvis.shore.mbari.org/thredds/dodsC/auv/dorado_ctd]
    Constraint: latitude[1:2:4000000]
    Error: An internal error was encountered in Vector.cc at line 587:
    The server sent declarations and data with mismatched sizes.
    Please report this to support@opendap.org

    notloaddap Brian Schlining checked a project into subversion that uses NetCDF-Java 4.0 as the data access library in matlab. Here's some examples based on the above data: >> ds = ncdataset('http://elvis.shore.mbari.org/thredds/dodsC/moorings/OS_M1_TS_090409') >> temp = ds.data('TEMP'); >> t = ds.data('TIME'); >> plot(t, temp) >> datetick('x', 2)

• Possible ways to address the above problems include
  1. Creating separate catalogs of the aggregated 10-minute data and the telemetered hourly data
  2. Creating an hourly gridded product from the 10-minute data and aggregating that with the hourly telemetered data, this would still be an irregular grid, but with the fewer points could be read by Ferret
  3. Submitting a bug report regarding the loaddap stride problem.

• BOG
  • Kevin has configured DRDS on his laptop and has demonstrated ability to query the data directly from the database:
     
    

• ROVCTD data
  • All of the ROVCTD data is in the Expd Relational database and can  be served with the same technique as we'll use for BOG.

Client tools for data access

• Ferret
• Matlab with loaddap
• Google Earth 5.0 as a geospatial-temporal data browser for data served through the THREDDS and other servers.

Results.jpg (image/jpeg)
QueryForm.jpg (image/jpeg)